Papers with sharpness-aware minimization
SharpSeq: Empowering Continual Event Detection through Sharpness-Aware Sequential-task Learning (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for continual event detection suffer from catastrophic forgetting . a novel continual learning paradigm leveraging sharpness-aware minimization is needed . |
| Approach: | They propose a continual learning paradigm that leverages sharpness-aware minimization and a generative model to balance training data distribution. |
| Outcome: | The proposed approach outperforms existing methods on real-world datasets. |
Revisiting Catastrophic Forgetting in Large Language Model Tuning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Catastrophic Forgetting (CF) compromises the effectiveness of large language models during fine-tuning, yet the underlying causes of CF remain largely unexplored. |
| Approach: | They propose a method to flatten the model loss landscape to mitigate CF by flattening the loss landscape. |
| Outcome: | The proposed method complements existing anti-forgetting strategies, further enhancing the resistance of LLMs to CF. |
Improving Sharpness-Aware Minimization with Fisher Mask for Better Generalization on Language Models (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning pretrained language models suffer from poor generalization . however, they add a perturbation to each model parameter equally, which is sub-optimal . |
| Approach: | They propose a sharpness-aware minimization optimization procedure that introduces a Fisher mask to improve the efficiency of SAM. |
| Outcome: | The proposed method outperforms the vanilla sharpness-aware minimization method on GLUE and SuperGLUE benchmarks. |
Train Flat, Then Compress: Sharpness-Aware Minimization Learns More Compressible Models (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in hardware, modeling, and optimization for deep neural networks have led to improvements in memory and inference efficiency. |
| Approach: | They propose to combine sharpness-aware minimization with various model compression methods to improve model compressibility. |
| Outcome: | Empirically, optimizing for flatter minima leads to greater compressibility of parameters compared to vanilla Adam when fine-tuning BERT models, with little to no loss in accuracy on the GLUE text classification and SQuAD question answering benchmarks. |
Sharpness-Aware Minimization with Dynamic Reweighting (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Deep neural networks are often overparameterized and can overfit training data. |
| Approach: | They propose an adversarial weight minimization algorithm that conducts adversarials and finds a common adversaria per-batch. |
| Outcome: | The proposed algorithm finds a common adversarial weight perturbation per-batch. |
Enhancing LLM Knowledge Learning through Generalization (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Continued pre-training on paraphrased data has shown empirical promise for enhancing knowledge acquisition, but this approach is costly and unreliable as it relies on external models or manual effort for rewriting. |
| Approach: | They propose formatting-based data augmentation which diversifies documents conveying the same knowledge by altering document formats rather than their content. |
| Outcome: | The proposed methods improve generalization to diverse paraphrased contexts and enhance pre-training and instruction tuning. |